Shore Side Data System (600031 and 900823) : Publishing other non-SIAM data to SSDS
This page last changed on Sep 30, 2009 by kgomes.

These are the instructions for putting non-SIAM infrastructure data streams into SSDS. There are two major steps for getting data into SSDS: Describing the data with metadata and Establishing a data publishing application.

Describe the deployments and data with XML metadata

Does not go through ingest

Note that this process does not send a JMS packet through the normal ingest chain. If you are SSDS savy, this means it will skip the transmogrify-ingest-ruminate step and put this information directly in the database.

  1. Devices that are sensors (things that make measurements) and instruments (things that produce data) must first be entered into SSDS so that the metadata author can use the SSDS unique Device IDs in the XML metadata. This may be done with the newDevice.jsp application, specifically:  http://new-ssds.mbari.org:8080/ssds/faces/newDevice.jsp.
  2. Construct the XML describing the platform, instrument, and sensor deployment. Using an XML schema-aware tool such as Oxygen is recommended for producing well-formed and valid XML. Below is an example XML file (1696.xml) for the Test deployment of the Eye In The Sea platform. Important things to note:
    1. A Deployment with role="platform" must be the outer element.
    2. Give the platform Deployment an appropriate name - this will appear in the SSDS Explorer application and may be used to find the data in SSDS
    3. Other attributes (startTime, nominalDepth, nominalLatitute, nominalLongitude) may be added to the platform Deployment element, though they may be added later to the SSDS Metadata database
    4. Specify the bufferItemSeparator, recordTerminator, and recordParseRegExp in the instrument Deployment RecordDescription to enable automated parsing of the output
    5. Specify the RecordVariables (the minimal attributes are shown in this example) 
      <?xml version="1.0" encoding="UTF-8"?>
<!-- $Header: /home/cvs/puckxml/1696.xml,v 1.4 2008/12/18 20:19:43 mccann Exp $	-->
<!-- Last edited by $Author: mccann $   -->
<Metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://ssds.mbari.org/xml/schema/SSDS_Metadata.xsd"
    majorVersion="1" minorVersion="2" lastAuthor="$Author: mccann $"
    lastUpdate="$Date: 2008/12/18 20:19:43 $">
    <Deployment role="platform" name="EITS on MARS (Test)">
        <Device id="1697"/>
        <!-- Eye In The Sea instrument for MARS2008 -->
        <Deployment role="instrument" name="Eye In The Sea combined data from the CTD and ADV">
            <Device id="1696"/>
            <Deployment role="sensor">
                <Device id="1694"/>
            </Deployment>
            <Deployment role="sensor">
                <Device id="1695"/>
            </Deployment>
            <output>
                <DataStream name="EITS Data Logger output of environmental data"
                    url="http://new-ssds.mbari.org:8080/servlet/GetOriginalDataServlet?deviceID=1696">
                    <RecordDescription bufferStyle="ASCII" bufferParseType="ordered"
                        bufferItemSeparator="whitespace" bufferLengthType="variable"
                        parseable="true" recordType="1" recordTerminator="\n"
                        recordParseRegExp="\s*([\-\d\.]+)\s*([\-\d\.]+)\s*([\-\d\.]+)\s*([\-\d\.]+)\s*([\-\d\.]+)\s*([\-\d\.]+)">
                        <RecordVariable name="Temperature" longName="Water Temperature"
                            units="deg C" columnIndex="1" format="float">
                            <StandardVariable name="sea_water_temperature"/>
                        </RecordVariable>
                        <RecordVariable name="Salinity" longName="Salinity" units="psu"
                            columnIndex="2" format="float">
                            <StandardVariable name="sea_water_salnity"/>
                        </RecordVariable>
                        <RecordVariable name="Depth" longName="Depth" units="meters" columnIndex="3"
                            format="float">
                            <StandardVariable name="Depth"/>
                        </RecordVariable>
                        <RecordVariable name="CurrentDirection" longName="Current Direction"
                            units="degrees magnetic" columnIndex="4" format="float">
                            <StandardVariable name="direction_of_sea_water_velocity"/>
                        </RecordVariable>
                        <RecordVariable name="VerticalCurrentVelocity"
                            longName="Upward Sea Water Velocity" units="m/s" columnIndex="5"
                            format="float">
                            <StandardVariable name="upward_sea_water_velocity"/>
                        </RecordVariable>
                        <RecordVariable name="HorizontalCurrentSpeed" longName="Sea Water Speed"
                            units="m/s" columnIndex="6" format="float">
                            <StandardVariable name="sea_water_speed"/>
                        </RecordVariable>
                    </RecordDescription>
                </DataStream>
            </output>
        </Deployment>
    </Deployment>
</Metadata>
  3. It's recommended that the final XML is to be checked into the puckxml module in MBARI's CVS on moonjelly.
  4. Submit the Metadata to SSDS using the SSDSLoads application (available at http://new-ssds.mbari.org/ssds-docs/client/. The -h option provides a usage note): 
    java -jar ssdsLoads-new-ssds.jar -d 1696.xml -x      # Use '-x' to test and then '-s' to submit
    1. If a mistake is made in the metadata (e.g. forgetting the platform Deployment) it may be easier to undo the submission and start over - do a deep delete on the parent deployment if this is the case.
    2. Minor attribute fixes may be done by editing the database once the deployments have been loaded
  5. Check that the metadata has been successfully loaded using the Explorer application.

Establish data publishing application

  1. This step assumes that there will be some application reading data from the deployed instrument. Perhaps it is a legacy application which reads the data to load into a custom data storage or visualization system. For this application to publish data to SSDS it must have visibility of each record the instrument produces as that record needs to be "packaged" into a SensorDataPacket and "published" to SSDS.
  2. An example Java application that will package and publish a record is below: 
    import java.io.IOException;
import moos.ssds.jms.PublisherComponent;
import moos.ssds.transmogrify.SSDSDevicePacket;

/**
 * <p>
 * Publish instrument data records to the SSDS database. The client must provide
 * SSDS device ID, timeStamp, sequence number, and payload.
 * </p>
 * <hr>
 *
 * @author : $Author: mccann $
 * @version : $Revision: 1.17.2.7 $
 *          <hr>
 *          <p>
 *          <font size="-1" color="#336699"><a href="http://www.mbari.org"> The
 *          Monterey Bay Aquarium Research Institute (MBARI)</a> provides this
 *          documentation and code &quot;as is&quot;, with no warranty, express
 *          or implied, of its quality or consistency. It is provided without
 *          support and without obligation on the part of MBARI to assist in its
 *          use, correction, modification, or enhancement. This information
 *          should not be published or distributed to third parties without
 *          specific written permission from MBARI.</font>
 *          </p>
 *          <br>
 *          <font size="-1" color="#336699">Copyright 2008 MBARI.<br>
 *          MBARI Proprietary Information. All rights reserved.</font><br>
 *          <hr>
 *          <br>
 */

/**
 * @author mccann
 *
 */
public class SsdsPublisher {

	/**
	 * Publish data record from an instrument to SSDS
	 *
	 * @param deviceID
	 * 			is the SSDS Device ID for the instrument that produces the data in payload
	 * @param epochMilliseconds
	 * 			time of payload sample in milliseconds since 1/1/1970 0000 GMT
	 * @param sequenceNumber
	 * 			an incrementing number for each packet
	 * @param payload
	 * 			data from instrument
	 */
	public static void publish(long deviceID, long epochMilliseconds, long sequenceNumber,
			String payload) {

		// Create a publisher component
		PublisherComponent pc = new PublisherComponent();

		// Create a new SensorDataPacket
		SSDSDevicePacket packetToSend = new SSDSDevicePacket(deviceID, payload
				.getBytes().length);

		// Set the packet type to data (0 = Metadata, 1 = Data, 2 = Message)
		packetToSend.setPacketType(1);

		// Assign the time
		packetToSend.setSystemTime(epochMilliseconds);

		// Set the parentID to 0 for a parentless deployment
		packetToSend.setPlatformID(0);

		// Set the metadataref number to 0, if the data format changes and we can
		// refer to a different metadata packet then this number will change.
		packetToSend.setMetadataRef(0);

		// Set the sequence number
		packetToSend.setSequenceNo(sequenceNumber);

		// Set the payload
		packetToSend.setDataBuffer(payload.getBytes());

		// Set the record type to 1 as this is the most common situation
		packetToSend.setRecordType(1);

		try {
			pc.publishBytes(SSDSDevicePacket
					.convertToPublishableByteArray(packetToSend));
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}

	}

	/**
	 * Test of SsdsPublisher.publish()
	 *
	 * @param args
	 *            No arguments are taken.  Test values hard coded.
	 */
	public static void main(String[] args) {

		/*
		 * Example packet for EITS instrument
		 */
		long eitsDeviceID = 1696; // 1696 is actual SSDS deviceID for EITS
		long sampleTime = 1228431283209L; // Sample time: 2008-12-04 22:54:43
		long sampleSequenceNumber = 1; // May set it to ID from DB insert

		// Payload values must match the RecordVariables in SSDS
		String samplePayload = "12.2, 4.3, 768.2, 2.3, 1.2, 0.2"; // Temp, Cond, Pres, U, V, W

		SsdsPublisher.publish(eitsDeviceID, sampleTime, sampleSequenceNumber, samplePayload);
	}

}
  3. An example of a shell script (the getM1 OASIS download) calling a perl script ssdsSubmit.pl to publish download statistics records to SSDS:
      
    # Record start of download
set starttime_es = `perl -e 'print time, "\n"'`

# Download the data and record error status
$DOWNLOAD $DOWNLOAD_OPTS  >& $DATAFILE
set rtnsts = $status

# Record the end of download
set endtime_es = `perl -e 'print time, "\n"'`

# Get the raw data filesize
set filesize = `ls -l $DATAFILE | sed -f $BIN/fs.sed`

# Log download statistics and submit to SSDS - IDs are unique for mooring deployment
set deviceId = 1698
set parentId = 1306
/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"
  4. If using Java incorporate the above class into a Java application that reads the data from the instrument and calls a method like the main() example for each record
  5. Set up the application to execute the loads for the duration of the deployment
  6. Configure DStoNetCDF.pl cron job execution as is done for the OASIS data processing within the ssdsadmin account on elvis. (For MARS deployments this is done for the DataProducerGroup MARS2008; see the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt/).

Generate a packet from a file and publish to SSDS

Under Construction

This section is still under construction, pay no attention

  1. Download a jar file
  2. Create the properties file like the one here: 
    # This file contains the information that will be used to construct a data packet to publish to SSDS.

# This is a SIAM property that (in theory) would allow multiple message formats
# that can be handled.  Right now, there is only one and it should always be 0.
DevicePacketVersion=0

# This is the ID of the source of the data (i.e. the device that generated the packet
# being sent.
SourceID=1

# This is the timestamp on the packet which normally represents when the packet
# was created.  In the system it is normally epoch seconds, but for clarity sake,
# you can enter it in ISO 8601 format (http://en.wikipedia.org/wiki/ISO_8601)
Timestamp=2009-09-30T15:47:00Z

# This is the sequence number that will be on the packet. It should reflect the
# index in the order of generation of packets from the source device.
SequenceNumber=1

# This is the reference number that points to the sequence number of the packet
# which contains the metadata that describes the contents of this packet. 
MetadataRef=0

# This is the ID of the parent device (if one exists) that the generating device
# (see SourceID above) was connected to when it generated the packet
ParentID=0

# This is the type of record that is being produced.  Source devices can produce
# multiple types of records of the same type. For example, a device can produce
# data records in multiple formats.  This allows you to identify which format 
# is used for the generated packet.  It serves to link the pre-defined metadata
# to the packet and allows machines and humans to understand the contents of the
# packet.
RecordType=0

# This value determines what type of packet (SIAM equivalent) will be constructed
# using the information in this file.  The available options are:
# 1. METADATA
# 2. SENSOR_DATA
# 3. DEVICE_MESSAGE
SecondStreamID=METADATA

# This indicates which version of the above type of packet will be constructed.
# At the time of this authoring, the only option available is 0 for all three
# types listed above. In theory this would allow you to have multiple forms of
# any of the above types of packets, but it has not been used to date.
SecondPacketVersion=0

# This is the name of the file (located with this properties file) that contains
# the payload of the first buffer.  It will be read as straight bytes into a
# byte buffer. NOTE: with METADATA packets the first buffer contains something
# that is called 'cause' which is not used much by SSDS.
FirstBuffer=test-first-buffer.txt

# This is the name of the file (located with this properties file) that contains
# the payload of the second buffer.  It will be read as straight bytes into a
# byte buffer. NOTE: with METADATA packets the second buffer is usually where the
# main metadata is stored (for example, device XML).
SecondBuffer=test-second-buffer.xml
  3. Create the two buffer files
  4. Run using command: 
    the command

Attachments:
ssdsSubmit.pl (text/html)
Document generated by Confluence on Feb 23, 2017 15:08